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The Beginning Teacher Evaluation Study, Phase II, was 
a research project on effective teaching behavior — what teachers do 
that significantly affects what and how pupils learn. The purposes of 
Phase II were to (1) develop an assessment system for measuring 
teacher and student behaviors and. other factors which could influence 
each of them and their interrelationships and (2) generate hypotheses 
about the interrelationships among teacher and pupil behaviors and 
related factors. Subjects were 41 second grade and 54 fifth grade 
experienced teachers in eight school districts in California. The 
APPLE Observation System was one of two observational systems used 
the study. The APPLE System involves a written record of the 
behaviors of teachers and pupils. "Scores" in the form of prorated 
frequencies from the observations were used as measures of teaching 
performance and pupil behavior. (RC) 
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The APPLE Observation System is a pupil-based system for recording 
behavior events in the classroom. Records of pupil behavior carry with 
them mandatory information describing pjjpil activities, the organizational 
structure of the classroom, the teacher ' s instructional activities and 
the teacher's response to the pupil behavior. Observers make English 
language descriptions of these events \dLth mandatory information vhich 
are subsequently encoded by reference to APPLE -Lexicons. The system 
is both a low-inference naturalistic observation process which retains 
the pupil behavior annotation, and also a system which yields categorical, 
observational data. ^-Jhen the observations are treated as a category, 
system, summaries of APPLE observation records measure *pupil behavior, 
pupil activit3^3 teacher activity, instructional context and teacher 
response using either the pupil or the classroom as the unit of 
anal^^sis. 

The computer software which was developed for the APPLE system' 
makes possible the combination of observation categories to yield 
measures of dyadic classroom variables such as combinations of Teacher 
Activity-Pupil Event, Pupil Event-Teacher Response or Teacher Activity- 
Teach^^r Response. In addition to behavioral records, the APPLE system 
incorporates a time recording feature so that it is possible to derive 
measures of duration of classroom behaviors over time. 

At the .completion of the 200 days of BTES observations, we encoded 
the APPLE records , summarized the observation categories by classroom, 
and computed the classroom allocation of time spent in instructional 
contexts for reading and mathematics. To develop performance measures 
for reading and mathematics instruction, we analyzed the pupil activity 
field to assign observations to either reading or mathematics lessons . 
Next we summarized the observation record for each classroom and 
developed measures of teacher activities, instructional contexts, 
pupil behavior and teacher responses which can be considered to be 
directly or indirectly related to teacher performance. 

The results of our preliminary examination of the relationship of 
these variables to student learning and attitude showed that performance 
does make a difference when the effects of the initial learning and 
attitude are controlled. 

An overview of the APPLE observation data showed that observed 
pupil behavior was a function of the type of instruction (reading or 
mathematics') the grade level of the pupil, the instructional context 
and the teacher activities. This finding is not surprising since cne 
would expect certain types of pupil behavior are appropriate for specific 
types of instruction and learning. 
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Wh2.1e pupil behavior varies from second to fifth grade, presumably 
due to differences in teaching activities and learning taskj teacher 
responses to pupils, and the quality of teacher responses to pupil 
behavior are remarkably similar from second to fifth grade. We infer 
that the ways teachers respond to children's behavior reflect stylistic 
and personality differences rather than differences attributable to the 
age of the children they are teaching. 

In organized instructional periods disruptive conduct problems and 
need for teacher discipline are not common and these variables were 
not sigiiif icantly related to pupil learning. This suggests that assessing 
teacher competence on the basis of the absence of behavioral problems 
during reading and mathematics instruction would be risky indeed. 

Classroom summaries of pupil characteristics which do make a 
diV^f^i^ence in reading are the frequencies of observed reading behaviors, 
and the frequency of on-task learning behaviors at second grade, and at 
fifth grade the frequency of observed reading behavior, evidence of 
participation in classroom activities and on-task behaviors. The 
frequencies of on-task pupil behaviors were significantly related to 
mathematics learning for both second and fifth grade. \^en we infer 
teacher performance from pupil behavior, we base it on the assumption 
that pupil behavior reflects the teacher's planning, instruction and 
management. 

Several teacher activities were associated with reading achievement 
outcomes at second grade. These could be described as the extent of 
direct instruction in reading as opposed to frequencies of observations 
of organizing the lesson *and explaining what the children were to do. 
Fewer teacher activity variables were significantly related to reading 
achievement at fifth grade presumably because fifth graders spend more 
time working independently of teachers at this level. 

Teacher feedback responses were associated with improved achievement 
in both reading and mathematics at both grade levels. 

Of considerable interest, and worthy of additional study, was the 
finding that using the classroom as the unit of analysis , -APPLE 
observation variables vjhich were significantly associated with reading 
and mathematics achievement were not necessarily those associated with 
positive attitude change. The fact that teacher performance variables 
which were related to signific^?nt increases in the criterion measures 
were specific to those measures, indicates that v;e must study teacher 
performance witli multidimensional, .outcome criteria. These are some o£ 
the highlights of the findings; .the Al^PLE observation system provides a 
rich source of data by v/hich to describe and analyze fieacher performance. 

Lambert, N. M. , Ilart£;ough, C. S. , Converse, F. L..and Converse, E. K, 
APPLE: An annotated pupil information system. Educational Technology, 
November 1971, 32-36. ^ 

Lambert, N, M. and Hartsough, Co S. APPLE observation variables and their 
relation to reading and mathematics achievement. Chapter 1, volum.e 3. 
Borannin p, Teacher Evaluation St udy. Princeton, New Jersey: Educational 
Testing Service, 1975, 
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Lambert, N. M. , Hartsough, C. S., Caffrey, C. and Urbanski, C. APPLE: 
Manual for APPLE Observers. Berkeley: University of California, 1976. 

Lambert, N. M. , Hartsough, C. S. , Caffrey, C. and Urbanski, C. APPLE: 

Lexicon for APPLE Observations . Berkeley: University of California, 1976. 
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APPLE Lexicon Definition of Selected 
Pupil Events 



+ ACADEIIC 

PERFOiUIAA'CE: 



+ ATTENTIVEN^SS: 



+ DIRECTION- 
FOLLOWING: 



ENGAGEMENT: 



+ P.\RTICIPAII02^: 



+ VOLUNTEERING: 



+ WORK HASITS; 

- COIJDUCT: 

- INATTENTION: 



- Ti\LKING: 
0 CONDUCT: 
0 EKGAGEM£NT: 



Level of performance in school tasks; response to assign- 
ments or lessons. Correct. 

Act of paying attention to the subject being taught or 
discussed, as shown through physical orientation, ges- 
tures, verbal response. Used only in positive sense and 
in situation where teacher is instructing or leading 
lesson. 



Behavior related to following directions given by the 
teacher. Cooperation with teacher directives, whether 
acadeaic or behavioral. 

Act of being involved in subject of lesson. Used v/hen 
pupil working independently of teacher. 

Description of an act of joining in a general class or 
group activity. 

Any spontaneous offering by the child to do a job or to 
give information, as in response to teacher's call for 
volunteers or to a question put to the entire group or 
class . 

Positive manner or nstliod of pursuing tasks, i.a. comple- 
tion of work, efficiency, organization, neatness. 

Negative school behavior which cannot be placed under a 
more specific evenu. 

Not paying attention to subject being taught or dis- 
cussed, as shovn throuv-fn ori-antation, vernal respomies, 
gestures. Usually implies a long-term distraction from 
activity bcin^ directed by teacher^ 

Inappropriate or forbidden speaking, such as speaking out 
of turn or during a quiet period. ' _ 

Neutral school behavior v/hich cannot 'be placed under a 
more specific event. 

Act of being engaged in subject of. lesson, but with less 
degree o.f involvcnent or attention than + EKGAGHMENT. 
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0 INATTENTION: 

0 WORX HABITS: 
+ ORAL READING: 

+ PHONIC SKILLS: 



+ READING 

COtlPREHENSlON: 

+ NUMBER CONCEPTS 



Pupil Events (Continued) 

Not paying attention to subject being taiasht or dis- 
cussed. Usually isiplies a short-term distraction from 
activity being directed by teacher. 

Neutral manner or method of pursuing tasks. 

Positive quality of students reading out loud; fluency 
accuracy, expressiveness, pacing, etc. 

Positive evidence of skill in sounding out new words; 
knowing the sound connected with letters. 

Pupil's correct understanding of what he has read. 

Positive knowledge of numbers and number relationships. 
Includes Identification of numerals, counting, skill 
with fractions, geometrical properties, etc. 



- NUMBER CONCEPTS: Absence of knowledge of numb 



0 NUI-IBER CONCEPTS ; 



......... ^xiowieage ot numbers and number relationships 

in comparison with age and class level. onsnips, 

MoicLT^^^^f^^ °! ""'"^"^ °^ relationships (e.g. 

concnntf ^ I understood) or absence of knowledge of 
concepts mare advanced than required by age or class level. 
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APPLE Lexicon Definition of Selected 
Contexts 



ADULT-GROUP: 



INDEPENDENT- 
CLASS: 



The class is divided into groups and an adult other than 
teacher is directing the activity of the group in which tlie 
child being observed is participating. 



The entire class is working or playing independently of the 
teacher's direction. Teacher usually present but not direc- 
ting. ' 



INDEPEIsDE^T^ 
GROUP: 



The group of students in which the child being observed is 
participating is working or playing independently of the 
teacher's direction. 



INDEPENDENTS- 
INDIVIDUAL: 



The student being, observed is working or playing independ- 
ently of the teacher's direction and also independently of 
the rest of the class. 



TEACHER- 
* CLASS: 



TEACIIER- 
GROUP: 



Teacher is directing the activity and the entire class is 
participating. 



The class is divided into groups and the teapher is direct- 
ing the activity of the group in which -the child being ob- 
served is participating. 
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APPLE Lexicon Definition of Selected 
Teacher Activities 



ACADmc 
ORGA>«ZING: 



Teacher 



ANSWERING: 



ASKING: 
AT DESK: 

AT BOARD: 
^CIIECKIKG: 

CIRCULATING: 



DISCUSSION: 
EXPLAINING: 
HELPING: 



INSTRUCTION 
GIVING: 



getting class or croup organized. Giving directions, 
explaining and doing other activities which direct the stu- 
dents into activities. 

Teacher answers question(s). May refer to a single answer to 
a single. question or to a general situation where teacher is 
answering questions put by several students over a period of 

llTlllsi'. "'""'"^ 'l-^tions of a single student, a group, or 

Teacher is doing work at desk. Used when teacher's activity 
IS unspecified or unknown and/or when she is available to 
help students at her desk. 

Teacher is working at blackboard, either alone (e.g., writing 
on board) or with student (s). . - • . ^ ^' * wricing 

Teacheris checking work to see if material is understood, 'to 

irhpfn^f ''^'"^ follo-..ed. to see how well work 
IS being done, or if work is correct. 

Teacher is circulating around room, interacting with stu- 
dents, asking and answering questions, giving help. Used 
^^vpr^°°''!•'°^^"^'' activity cannot be identified or when 
ITclf f '^i^^'^^" happening one after the other, too " 

quickly to be specific. 

Teacher is leading a discussion. Eniphasis on student res- 
ponse and comprehension, not sinply on facts. 

Teacher is explaining something, eithef in response to a 
question or as part of a process of giving instructions. 

Usaf S '"^ ^ ^"^^^ " P"^^^^ unspecified manner. 
Used v.hen more precise activity, such as explaining or ans- 
wering cannot be distinguished, or ...;haP. several helping 
•activities are happening t^o quickly to be specif ied^ ^ 

tivft\'^J%f;''^"'/"'''""''°"' '-'^""'^ ^^^-^Sned ac- 

tivity 01 about classroom organization. " 
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Teacher Activities CCoiAinueri) 



LISTENING: 



QUESTION AND 
ANSWER: 



SUPERVISING: 



Teacher listens to a studei:!.. either during oral reading or 
while student speaks to her or asks a question. 



Teacher is asking a series of qviestions, which students ans- 
wer. Used prinariljr In group or i.lass context and d/.stin- 
guished from ASKING by serial nature of questions. 

General situation where teacher is watching over class or a 
group. May occasionally ask or answer questions, but gen- 
eral emphasis is on observation rather than interaction 
With students. 



WORKING WITH: 



General unspecified instructional activity of working with 
an individual, a group or the class. Used when no other 
teaching activity is specified. 



9 



ERIC 



INSTRUCTING: 



DISCIPLINING: 



TEACHER HELP: 



IGNORING: 



MOVES ON: 



NEGATIVE 
FEEDBACK: 



NONE: 



POSITIVE 
FEEDBACK: 



PRAISE: 

QUESTIONING: 

RECOGNIZING: 

KKDTRECTING: 



APPLE Lexicon Definition of Selected 
Teacher Responses * 

Teacher activities connected vith her traditional jobs of 

instructing end supervising the everyday business of the 

classrooa "gave a lesson/' directed, organized, "called 
on pupil," answered a question. 



t to 



Punishment imposed on child for his behavior, e.g. sent 
principal's office or told .to sit in corner or to stay 
after school. 

Teacher gives some specif ic instructional assistance to' an 
individual student. • 

Teacher seecdngly av/are but not responding to behavior or 
performance of pupil. 

Teacher does not comnent on or give feedback to a pupil 
response, as in a question and answer session; instead, she 
calls on another pupil or moves on to another part of the 
lesson. Can occur when pupil's response is correct or in- 
correct. 



Mild negative verbal reinforcement in response to academic 
or behavioral performance. e.g., "wrong,' 



no . 



No response to pupil perfbrruance or behavior. No distinc- 
tion made between whether teacher aware of this performance 
or behavior or not (unless is clearly a situation of ' 
IGNORING) . 



Use of nild positive verbal reinforcement in response to 
academic or behavioral performance, e.g., "right "0 K " 
•'good," "fine," "correct." ' 

Teacher commends or tangibly rewards a student's activities 
or products . 

Teacher responds to pupil's behavior or answer to a previ- 
ous question by hers^Ui asking a question. 

Teacher's neutral nonevaluativc response to an academic 
performance or a nonacademlc beimvior. ' e.g., "notices", 
"acknoi:lodr;c*;" . 

A verbal or non-verbal action by the" teacher to redirect a 
pupil's behavior. Done without: apparent an^cr. 
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COMBINED APPLE OBSERVATION CATEGORIES* 



PUPIL EVE>rrs 
Apple 1 



+ Oral Reading 
+ Phonic Skills 
+ Reading Comprehension 

Apple 2 luAfr^/ueai o Conduct 

^Jpp -f]fV-CJS. ° Engagement 

0 Inattention • 
0 Work Habits 



Apple 3 



+ Attentiveness 

+ Direction Pollowing 

+ Engage; it 

+ Particli^ation 



Conduct 

Inattention 

Talking 



+ Volunteering 

+ Work Habits 

.+ Academic Performance 



TEACHING CONTEXTS 



Apple 4 
Apple 5 

Apple 6 
Apple 7 



Adult^Group 

Ind ep end en t-*Clas s 
Independent -Group 
Independent-Individual 

Teacher-Group 

Teacher-Clas9 



TEACHER ACTIVITIES 



Apple 8 



Apple 9 



Apple 10 



Apple 11 hO0l»-iTV 



Academic Organizing 
Helping 

Instruction Giving 
Supervising 
Working With 

Answering 

Asking 

Checking 

Discussion 
Explaining 
Listening 

Question and Answering 

At Desk 
At Board 
Circulating 

11 
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COMBINED APPLE OBSERVATION CATEGORIES 
(continued) 



TEACHER RESPONSES 

Apple 12 Instructing Teacher Help 

Questioning Recognizing 
Positive* Eaedback 
Praise 

Apple 13 Disciplining 

Redirecting 

Apple 3 4 . Ignoring 

Negative Feedback 
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*In the Beginning Teacher Evaluation Study, APPLE Observation variables were 
combined into logical groupings for the structural and regression analyses • 
The composition of the 14 combined APPLE variables is listed in this table. 
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